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PREFACE 


This  report  documents  the  process  and  results  from  a  feasibility  study  and  evaluation  of 
computer-based  occupational  data  collection.  The  study  described  herein  was  accomplished  for 
the  Air  Force  Occupational  Measurement  Squadron  in  collaboration  with  the  Human  Resources 
Directorate  of  the  Armstrong  Laboratory  and  the  Royal  Australian  Air  Force.  The  feasibility 
study  was  conducted  to  assess  the  operational  and  logistical  problems  involved  with  the 
development,  implementation,  and  evaluation  of  computer-based  job  and  occupational  data 
collection  methods. 

Earlier  versions  of  this  paper  were  presented  at  International  Military  Testing  Association 
conferences  in  1994, 1995,  and  1997.  We  gratefully  acknowledge  the  technical  support  provided 
by  the  Air  Force  Occupational  Measurement  Squadron.  We  also  acknowledge  the  critical  role 
the  Air  Force  Career  Field  Managers  of  the  specialists  used  in  this  study  played  in  its  successful 
completion. 
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Introduction  and  Overview 


Introduction 

The  Air  Force  has  been  studying  the  possibility  of  automating  administration  of 
occupational  surveys  (Albert,  Phalen,  Selander,  Yadrick,  Weissmuller,  Dittmar  &  Tucker,  1 99  j  , 
Albert,  Phalen,  Selander,  Dittmar,  Tucker,  Hand  Weissmuller  &  Rouse,  1994).  Preliminary 
results  from  experimental  studies  suggested  that  the  use  of  computers  for  occupational  surveys 
was  highly  possible  and  the  quality  and  reliability  of  data  collected  might  be  substantially 
improved  (Mitchell,  Weissmuller,  Bennett,  Agee,  &  Albert,  1995;  Phalen  &  Mitchell,  1996).  A 
field  study,  conducted  in  1994  and  1995,  examined  the  feasibility  of  computer-based  survey 
administration  using  Air  Force-developed  software  and  normal  operational  administration 
procedures. 

This  report  documents  a  feasibility  study  conducted  by  the  Institute  for  Job  and 
Occupational  Analysis  (I  JO  A)  for  the  Occupational  Survey  Branch  of  the  Air  Force 
Occupational  Measurement  Squadron  (AFOMS/OMY)  in  collaboration  with  the  Human 
Resources  Directorate  of  the  Armstrong  Laboratory  and  the  Royal  Australian  Air  Force.  The 
objectives  of  this  study  were  to  conduct  a  feasibility  study  of  the  development,  implementation, 
and  evaluation  of  computer-based  job  and  occupational  data  collection  methods.  The  study 
included  an  evaluation  of  the  quality  of  the  data  obtained  from  both  a  paper-and-pencil 
occupational  inventory  and  a  computer-based  version  of  the  same  inventory.  Additionally,  the 
study  assessed  the  impact  of  logistics  associated  with  automating  the  process  -  not  with  the 
particular  software  employed.  Finally,  the  study  projected  costs  and  schedules  for  automated 
surveys  based  on  the  observed  ease  of  deployment  in  this  study  and  identified  implementation 
issues  which  may  be  associated  with  computer-based  development  and  delivery  systems. 

Specific  objectives  of  the  study  were  as  follows: 

(1)  Develop  a  detailed  feasibility  study  and  analysis  plan; 

(2)  Conduct  and  document  a  critical  review  of  existing  computer  survey  capabilities 
and  commercial  products: 

a.  Conduct  a  review  of  Air  Force  bases  and  Survey  Control  Monitors 
(SCMs);  this  review  included  contacting  bases  in  the  continental  United 
States,  Pacific  Air  Forces  (PACAF),  and  United  States  Air  Forces,  Europe 
(USAFE)  to  determine  the  extent  and  capabilities  of  available  computer 
resources  to  support  data  collection. 

b.  Review  existing  commercial  job,  occupational,  and  position  analysis 
methods  and  recommend  a  method  for  use  in  the  feasibility  study.  For  this 
review,  criteria  such  as  cost  of  obtaining  each  method,  proprietary 
restrictions  on  the  use  of  the  method,  capacity  of  the  method  to  accept  AF 
task  lists,  size  of  the  program  for  data  collection  (i.e.,  desired  method  shall 
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include  execution  and  data  collection  on  a  single  International  Business 
Machines/Personal  Computer  (IBM/PC)-compatible  diskette  for  each 
survey  type),  ease  of  data  collection  and  transfer  to  a  larger  data  file,  and 
the  capability  of  the  method  to  collect  job  and  occupational  data  from 
respondents  using  available  PC-size  computers,  not  merely  to  score  data 
obtained  from  a  scanable  sheet  or  from  other  non-electronic  media. 

(3)  Identify  two  AF  specialties  (AFSs)  for  which  paper-and-pencil  occupational 
inventories  have  been  developed  and  fielded.  These  AFSs  will  serve  as  the 
candidate  AFSs  for  the  development  and  feasibility  test  of  the  computer-based 
method. 

(4)  Review  and  debug  the  computer-based  versions  of  the  job  inventory  and  the  task 
factor  survey  for  each  AFS.  AL/HR  researchers  will  develop  versions  of  each 
survey  and  provide  them  to  AFOMS  for  use  in  the  project.  Development  activities 
include: 

a.  Downloading  each  AFS  task  list  from  the  AFOMS  computer,  and 

b.  Configuring  cover  pages,  respondent  instruction,  and  task  lists  to 
approximate  the  paper-and-pencil  version  of  the  inventory  for  delivery  and 
data  collection  of  each  different  type  of  survey  on  a  single  diskette,  and 
using  a  unique  survey  control  number  convention  to  enable  computer- 
based  survey  to  be  distinguished  from  the  paper-and-pencil  survey  data. 

(5)  Select  equivalent  random  samples  of  field  personnel  to  receive  the  computer- 
administered  and  paper-and-pencil  surveys.  AFOMS  distributed  the  computer- 
based  versions  to  field  personnel  identified  in  the  sample  using  their  established 
occupational  survey  distribution  process. 

(6)  Apply  criteria  for  the  feasibility  test  and  evaluation  of  the  computer-based  method 
and  the  traditional  paper-and-pencil  method  and  associated  data.  These  criteria 
included: 

a.  Document  costs  associated  with: 

(1)  development  (e.g.,  time  and  materials), 

(2)  delivery  (e.g.,  mailing,  time,  and  materials), 

(3)  data  retrieval  and  use, 

b.  Identify  logistics  issues  and  constraints: 

(1)  distribution  to  any  work  location, 

(2)  printing, 

(3)  disk  copying, 

(4)  computer  availability  in  the  field, 
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(5)  ease  of  computer  use  by  respondents,  and 

(6)  ease  of  data  aggregation  from  single  inventory  to  aggregate  data 
files  and  from  single  diskette  to  aggregate  data  file. 

c.  Assess  data  quality 

(1)  return  rates, 

(2)  representativeness  of  data  from  respondents  using  both  methods, 
group  membership,  comparable  cases,  diagram,  and 

(3)  interrater  agreement  of  responses,  and  comparable  background 
data. 

d.  Compare  specific  data  elements  for  each  survey  type 

(1)  time  spent  ratings, 

(2)  percent  members  performing  task  ratings,  and 

(3)  task  factor  data  (task  difficulty  and  training  emphasis  ratings). 

e.  Gather  user  responses  related  to  surveys 

(1)  reactions  to  using  the  computer-based  survey  method  compared  to 
paper-and-pencil  methods, 

(2)  ease  of  survey  response  by  users  of  both, 

(3)  convenience  of  survey  approach  and  data  collection,  and 

(4)  availability  of  various  levels  of  computer  technology  (disk  drive 
size,  mouse,  color  monitor,  etc.). 

Returned  diskettes  were  forwarded  from  AFOMS  to  IJOA  (as  the  complete  data  files  developed 
from  the  paper-and-pencil  inventory  responses).  IJOA  performed  virus  checking  and  data 
aggregation  for  diskette  surveys  and  subsequently  compared  the  data  obtained  from  each  type  of 
inventory. 

(7)  Provide  a  final  report  documenting  the  study  results  and  recommendations  of  this 
work  to  the  Air  Force. 
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Technology  Review 


Review  of  the  Literature 

Previously  when  conducting  surveys  using  a  computer  for  administration,  specialized 
computer  skills  and  expensive  computer  equipment  were  required.  However,  now,  with  the 
increasing  sophistication  in  the  personal  computer  arena,  computer  administered  surveys  in 
organizational  settings  are  becoming  a  favorable  alternative.  Rosenfeld,  Booth-Kewley,  and 
Edwards,  1993,  list  four  advantages  to  administering  a  computer  survey:  1)  survey  data  entry  is 
automated,  2)  missing  data  can  be  eliminated,  3)  there  are  no  out-of-range  responses,  and  4) 
complex  item  branching,  transparent  to  the  respondent,  can  be  used.  Because  the  survey  data  are 
automated,  the  chance  for  errors  to  occur  during  data  collection  or  data  analysis  is  minimal. 

Also,  processing  time  for  data  analysis  is  greatly  reduced  (Rosenfeld,  et  al,  1993).  The  logistic 
and  technical  requirements  of  large-scale  paper  surveys  typically  require  a  year  or  more  from 
start  to  finish.  Therefore,  the  findings  are  often  outdated  by  the  time  they  are  obtained  (Edwards, 
Rosenfeld,  Booth-Kewley,  &  Thomas,  1996). 

Most  of  the  comparative  reviews  of  job  analysis  methods  (e.g.,  McCormick,  1979;  Gael, 
1988;  and  Levine,  1983)  do  not  include  any  information  on  computer-based  surveying  systems. 
A  recent  survey  of  job  and  occupational  analysts  in  Australia,  Canada,  and  the  United  States 
indicates  that  computer-based  surveying  is  among  their  highest  priority  interests  for  near-term 
research  and  development  (Fugill  &  Weissmuller,  1993). 

Computers  are  often  used  for  analysis  of  data.  The  data  are  typically  scanned  from 
optical  or  mark-sensitive  score  sheets.  Data  may  also  be  Used  for  interfacing  with  a  cumulative 
database  or  set  of  norms,  as  with  the  PAQ/PMPQ  Enter- Act  system  (PAQ  Services,  1 990).  This 
permits  a  generation  of  norm-based  individual  and  group  analysis  reports.  Other  job  analysis 
systems,  such  as  the  Fleishman  Job  Analysis  System  (F-JAS),  use  scanable  score  sheets  and 
suggest  that  computer  administration  will  be  developed  sometime  in  the  future  (Fleishman  & 
Reilly,  1992). 

The  most  recent  comprehensive  review  of  job  analysis  (Harvey,  1991,  71-163)  highlights 
the  need  for  development  of  computer  administration  for  job  analysis  functions.  Harvey  (1991) 
indicates,  that  part  of  the  resistance  to  the  collection  of  needed  comprehensive  task  information 
has  been  the  relative  high  cost  of  such  data  collection.  Harvey  suggests  that  "advances  in 
computer  hardware  and  software  technology  hold  the  promise  of  significantly  reducing  the  cost 
and  labor-intensiveness  of  this  approach  to  job  analysis"  (p.  117).  Harvey  also  notes  that: 

Today  desktop  personal  computers  costing  under  $10,000  can  match  the  performance  of 
the  million-dollar  mainframe  systems  that  only  1 5  years  ago  were  required  to  run 
programs  like  CODAP.  Thus,  the  computing  power  required  to  manage  job  analysis  data 
bases  is  within  the  reach  of  even  the  smallest  organization.  Efforts  to  develop 
computerized  integrated  personnel  systems  (IPS)  have  increased  dramatically  in  recent 
years  (e.g.,  Avner  &  Mayer,  1986;  Harvey,  1986;  Mitchell  &  Driskill,  1986;  Wilson, 
1987). 
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Although  computerized  integrated  personnel  systems  can  make  the  process  of  managing 
and  using  large  amounts  of  task-based  job  analysis  data  practical  for  both  large  and  small 
organizations,  the  problem  of  collecting  the  task  information  remains  (Harvey,  1991, 

117). 

Harvey  (1991)  recommends  the  development  of  Artificial  Intelligence  (Al-based  systems 
to  assist  in  the  development  of  task  inventories  and  other  job-related  information.  He  reports  the 
development  of  job  analysis  computer  software  by  Wilson  (1991)  for  capturing  ability 
requirements  ratings,  which  presumably  would  use  a  single  standardized  task  list.  Wilson 
indicates  he  used  an  AI  shell  to  capture  raters'  responses  to  a  set  of  questions;  he  found  it  took 
about  the  same  amount  of  time  as  traditional  paper-and-pencil  ratings,  but  that  respondents 
seemed  to  prefer  the  computer  administration  (Mark  Wilson,  personal  communication,  24  May 
1994).  Wilson  &  Zalewski  (1994,  p.  200)  note  that  "we  find  it  surprising  that  little  research  has 
focused  on  the  application  of  expert  system  technology  to  collecting  information  about  jobs  (job 
analysis)." 

Harvey  (1991)  recommends  an  "alternative  approach"  for  reducing  the  cost  of  collecting 
detailed  task  information  through  use  of  a  "generic  task  inventory"  for  broad  "occupational 
categories  (e.g.,  clerical,  managerial,  etc.)"  based  on  a  synthesis  of  prior  job  analysis  research 
(Harvey,  1991).  He  concludes  his  discussion  of  this  area  with  the  comment: 

Although  standardized,  worker-oriented  questionnaires  are  useful  for  many  personnel 
functions,  there  are  some  applications  of  job  analysis  data,  such  as  development  of 
training  programs,  that  can  only  be  solved  with  task-oriented  job  analysis  data:  Making 
such  methods  cost  effective  is  a  critical  goal  (Harvey,  1991,  p.  118). 

In  an  effort  to  determine  the  differences  between  computer  and  paper-and-pencil 
administration  of  noncognitive  measures,  Potosky,  and  Bobko  (1997),  looked  at  the  redesign  of 
the  system  to  select  Air  Traffic  Controller  Specialists  (ATCS)  in  the  United  States,  for  the 
Federal  Aviation  Administration  (FAA).  Interested  applicants  took  both  a  computer  and  paper- 
and-pencil  version  of  the  ATCS  selection  measures.  This  study  indicated  that  for  the  empirical 
(raw)  cross-mode  correlations  were  greater  than  .90,  and  the  estimated  latent  cross-mode 
correlations  were  all  1.0.  This  suggests  that  one  can  administer  these  noncognitive  measures  by 
means  of  computer  with  considerable  confidence  (Potosky  &  Bobko,  1997). 

Telephone  surveys  also  have  been  enhanced  through  the  use  of  computer-assisted 
telephone  interviewing  (CATI)  systems.  The  CATI  system  operates  by  allowing  the  computer  to 
control  the  correct  order,  and  branching  of  the  questions  when  necessary.  The  CATI  issues  error 
messages  when  responses  are  out  of  range,  and  stores  the  responses  in  a  data  base  for  future  data 
analysis  (Edwards,  et  al,  1993).  This  system  has  provided  gains  in  efficiency  and  error 
reduction. 

This  literature  review  indicates  computer  administered  survey  techniques  are  a  becoming 
of  increasing  interest  due  to  the  benefits  of  time,  cost,  and  reduction  of  administration  and 
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analysis  errors.  Mitchell,  et  al  (1994),  indicate  also  that  "respondents  appear  to  have  little  if  any 
difficulty  completing  the  surveys  on  PCs,  and  their  attitudes  are  for  the  most  part  extremely 
positive."  During  computer  administration,  disks  can  be  produced  faster  and  a  considerable  cost 
savings  when  compared  to  precision-printed  optically  scanable  forms.  In  order  to  possibly  see 
these  discussed  advantages  one  should  consider  using  computer  assisted  techniques  for  job  and 
occupational  data  collection  methods  if  feasible  for  the  particular  task  at  hand. 

Review  of  Available  Commercial  Software 

For  maximum  flexibility,  to  avoid  licensing  issues  and  the  payment  of  royalties,  it  was 
considered  desirable  to  use  government-developed  and  -owned  software.  Unfortunately,  at  the 
outset  of  this  project,  all  candidate  government  systems  were  "research  tools"  rather  than 
certified  occupational  survey  engines.  With  that  in  mind,  a  temporary  license  of  a  commercial 
product  was  considered  a  possibility,  especially  if  one  wanted  to  guarantee  a  stable  product 
where  software  bugs  would  not  disrupt  return  rates  used  to  evaluate  overall  automated  logistics. 

One  of  the  problems  with  use  of  commercial  systems  is  the  license  agreement  which  sets 
out  fees  and  restrictions  on  use.  Commercial  software  systems  are  protected  by  copyright  laws 
and  cannot  be  used  except  in  accordance  with  permissions  of  the  copyright  holder  and  use  incurs 
a  "per-use"  or  site  license  fees.  This  limits  the  desirability  of  commercial  software  for 
widespread  Air  Force  use  or  for  transfer  to  other  AF  organizations  for  use  in  small,  specialized 
applications. 

Another  of  the  problems  with  use  of  commercial  systems  is  the  fact  that  one  cannot 
modify  (adapt)  a  proprietary  system.  Should  a  commercial  product  closely  meet  the  Air  Force's 
needs,  it  may  be  more  cost  effective  to  obtain  source  code  and  unlimited  rights  from  the 
commercial  firm.  While  this  is  a  possibility,  it  is  important  to  remember  that  the  present  study 
sought  to  elaborate  logistical  issues  in  automated  surveying  and  was  not  aimed  at  selecting  an 
automated  product. 

Four  commercial  products  were  identified  and  evaluated.  These  four  survey  engines 
included  MicroSURV,  Sawtooth,  atSURVEY,  and  RaoSoft  each  of  which  are  reviewed  below. 

Following  the  review  of  commercial  systems  is  an  overview  of  government-  developed 
and  -owned  software. 

MicroSURV  -  A  review  of  recent  catalogs  of  major  test  and  job  analysis  publishing  firms 
(Consulting  Psychologists  Press,  1994;  The  Psychological  Corporation,  1994)  indicates  that  most 
firms  are  not  currently  marketing  computer-based  job  analysis  systems;  typically  their  products 
are  pencil-and-paper  forms  with  scanable  score  sheets  (such  as  Psychological  Corporation’s  new 
Common  Metric  Questionnaire).  However,  the  1 994  Catalog  for  the  Assessment  Systems 
Corporation  ("World  Leader  in  Computerized  Testing")  includes  announcement  of  a  new  system 
called  MicroSURV  ("The  Microcomputer  On-Line  Survey  System"). 
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This  PC-based  system  is  designed  primarily  for  data  collection  and  is  purported  to  have 
mapping/branching  capabilities.  It  also  includes  a  time/clock  feature  to  document  survey  time 
and  a  cutoff  feature  to  automatically  cut  off  administration.  The  system  can  deliver  either 
multiple  choice  or  Likert-type  question  formats.  MicroSURV  can  handle  up  to  400  items 
[Assessment  Systems  Corporation  (ASC),  1994,  p.  15];  this  limitation  rules  this  system  out  for 
most  occupational  surveys,  based  on  the  size  of  typical  task  lists  and  background  sections.  In 
any  case,  the  system  is  advertised  as  "Available  Fall  '94"  and  thus  is  not  yet  available.  Costs  are 
also  not  yet  determined.  Equipment  requirements  for  this  system  are:  IBM  PC  or  100% 
compatible  computer;  DOS  (3.3  or  higher),  and  640  Kb  RAM.  A  hard  disk  is  required  for 
questionnaire  development.  There  are  some  conversion  programs  available,  according  to  the 
ASC  catalog,  which  will  make  ASC  DOS-based  products  useable  on  Macintosh  equipment  (p. 

4). 


Sawtooth  -  Sawtooth  Software  (of  Evanston,  IL  and  Ketchum,  ID)  has  fielded  a 
computer-aided  telephone  interviewing  and  analysis  system,  which  also  has  capabilities  for 
adaptive  conjoint  analysis,  adaptive  perceptual  mapping,  and  cluster  analysis  (Sawtooth 
Software,  1989).  Of  interest  here  is  their  "Disks-by-Mail"  system  which  facilitates  data 
collection  by  mailing  a  survey  to  individuals  to  be  accomplished  on  their  personal  or  business 
PCs.  The  advantages  of  this  system  include  standard  statement  of  questions  (versus  telephone 
interviewing),  higher  response  rates,  and  little  requirement  to  scan  and  "clean  up  data  (Sawtooth 
Software,  1989,  pp.  4-5).  Practical  considerations  include  difficulties  in  deciding  the  size  and 
densi  ty  of  disks  to  mail,  concern  with  computer  viruses,  and  potential  compatibility  problems 
with  some  types  of  monitors  and  printers.  Software  for  creating  surveys  is  available  for  purchase 
(up  to  100  variables  for  $500,  up  to  1000  variables  for  $4,000).  A  "variable"  is  any  possible 
choice  for  any  question;  thus  the  1000  variable  upper  limit  may  be  a  problem  for  accomplishing 
military  occupational  surveys,  which  contain  large  numbers  of  background  items  in  addition  to 
300  -  1 800  tasks.  Purportedly,  the  source  code  for  the  "Disks-by-Mail"  and  clustering  programs 
could  be  made  available  to  the  government  for  $25,000  for  modification  and  use  (Sawtooth  letter 
to  Mr.  Walter  Albert,  1991). 

atSURVEY  -  Another  commercially-available,  computer-based  survey  system,  and  one 
designed  specifically  for  the  task-based  approach  of  job  analysis,  is  the  atSURVEY  system. 
atSURVEY  is  an  ancillary  product  associated  with  atCODAP,  a  micro-computer  based  version 
of  CODAP,  developed  by  Sensible  Systems,  Inc.  of  San  Antonio,  Texas  (Staley,  Weissmuller, 
Lewis,  &  Johnson,  1987;  Weissmuller,  Staley,  Lewis,  &  Johnson,  1987).  atSURVEY  has  been 
tested  on  a  variety  of  DOS-based  platforms  (see  Sage,  1993).  The  atSURVEY  system  was 
employed  recently,  using  3.5"  720  Kb  disks  which  were  mailed  to  14,000  Ford  Motor  Company 
engineers  in  North  America  and  5,400  engineers  in  Ford's  European  Automotive  Operations. 
This  was  a  very  successful  application  which  resulted  in  meaningful  data  for  use  by  Ford 
management  to  identify  training  requirements  (Tsai,  1993,  p.  12).  A  site  license  for  atCODAP  is 
$50,000  for  the  first  year  with  $5,000  renewal  for  each  subsequent  year.  This  atCODAP  license 
with  all  standard  analysis  capabilities  (such  as  interrater  reliability,  case/task  clustering,  job 
descriptions,  etc.)  includes  an  unlimited  world-wide  license  for  atSURVEY. 
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Because  the  atSURVEY  system  is  used  for  occupational  analysis,  it  offers  a  reasonable 
template  for  reviewing  the  entire  automated  survey  process.  The  atSURVEY  system  is  divided 
into  five  phases. 

Phase  One:  Survey  Creation  -  Phase  One  involves  actually  creating  the  inventory. 
atSURVEY  accepts  flat  ASCII  files  from  any  word  processor/text  editor  as  well  as  input  from 
standard  sources  such  as  ASCII  CODAP  (Unisys,  VM,  MVS,  RISC),  FIELD  AT A  CODAP,  IBM 
CODAP  360,  CODAP/OSA  (Occupational  Surveys  Australia,  LTD,  Unisys),  and  TIARA. 

When  inputting  text  or  converting  and  modifying  previous  inventories,  a  suite  of  audit  programs 
is  available  to  ensure  proper  format  prior  to  mastering  a  survey  disk. 

Phase  Two:  Disk  Mastering  -  Phase  Two  is  the  mastering  of  a  survey  disk.  When  located 
in  the  directory  with  the  text  files  describing  the  inventory,  insert  a  freshly  formatted  diskette  and 
type  “atDisk  A:”  or  “atDisk  B:”  to  create  a  Master  on  the  desired  drive  (and  disk  format). 

Phase  Three:  Disk  Reproduction  -  Reproducing  the  master  disk  in  the  proper  quantities 
and  proper  sizes  (5.25"  x  360Kb;  5.25"  x  1.2Mb;  3.5"x720Kb;  and  3.5"xl.44Mb)  is  required. 
Orders  for  500  to  5000  disks  are  typically  reproduced  and  returned  within  four  days.  Costs  run 
around  $1 .00  per  disk  with  customized  professional  labels  adding  about  $0.04  per  disk. 

Phase  Four:  Survey  Administration  -  This  phase  involves  targeting,  shipping,  local 
distribution/collection  (if  desired),  return  shipping,  and  return  shipment  inspection  and  log-in  of 
actual  survey  disks. 

Phase  Five:  Data  Aggregation  -  Data  aggregation  includes  preliminary  quality  control, 
data  base  initialization  (selection  of  a  target  hard  disk  directory),  and  incorporation  of  contents 
from  each  disk  using  the  atLOAD  program.  atLOAD  produces  an  atCODAP-ready  data  base.  A 
conversion  program,  EXPORTCD,  converts  the  atCODAP  Case  Data  File  into  ASCII  CODAP, 
card-image  format. 

Discussion  -  Although  the  U.S.  Coast  Guard  (USCG)  is  a  licensed  user  of  the  atCODAP 
system  for  inventory  development  and  analysis  of  data,  the  USCG  has  not  yet  attempted 
automated  survey  administration  (Lanterman,  personal  communication,  24  May  1994).  Their 
internal  PC  capability  uses  a  unique  operating  system  which  is  not  compatible  with  DOS  or 
Macintosh  systems.  Ultimately,  the  USCG  occupational  analysis  program  would  prefer  PC 
survey  administration,  with  inventories  and  specific  individual  responses  transmitted  by  E-mail 
(Lanterman,  personal  communication,  24  May  1994). 

Licensed  use  of  the  atSURVEY  system  by  itself  can  be  obtained  from  Sensible  Systems, 
Inc.  for  a  fee  of  $1 .00  per-disk;  this  does  not  cover  any  costs  to  set  up  the  inventory  or  reproduce 
the  disks.  This  fee  permits  Air  Force  (or  IJOA)  personnel  to  use  the  necessary  software  required 
to  survey  the  number  of  individuals  contracted.  A  program,  EXPORTCD,  permits  the 
reformatting  of  the  final  data  base  (from  atLOAD)  for  use  in  ASCII  CODAP  (Unisys,  VM, 

MVS,  or  RISC). 
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RaoSoft  Survey  -  In  1994,  the  U.S.  Army  occupational  survey  staff  began  the 
“reengineering*’  of  the  Army  Occupational  Survey  Program  (AOSP),  now  retitled  as  the 
Occupational  Data  Analysis,  Requirements,  &  Structure  (ODARS),  at  the  direction  of  the  Army 
Research  Institute  (ARI),  to  which  the  AOSP  is  now  assigned  (Worstine,  1995).  The  use  of  this 
commercial  software  (about  $300  per  copy;  $395  for  site  license  which  allows  for  data  collection 
on  disks  (p.  1 1 8)  appears  to  be  focused  on  obtaining  training  evaluation  data  for  the  various 
Army  schools,  rather  than  full  field  task  inventory  administration  for  multiple  users.  This  new 
impetus  appears  to  be  a  result  of  ODCSSPER  or  ARI  mandate  that  survey  time  (survey 
development  and  field  data  collection)  must  be  reduced  to  six  months  (p.  97). 

While  the  RaoSoft  system  has  a  number  of  attractive  features,  it  presently  does  not  have 
the  capability  to  reformat  data  into  CODAP  standard  input  files,  process  the  typically  large  task 
data  sets,  nor  meet  the  requirements  for  field  administration  flexibility  directed  on  the  present 
feasibility  study.  Under  a  recent  Army  procurement,  RaoSoft  Incorporated  will  add  case 
clustering  and  inter-rater  reliability  capabilities  to  their  software  for  the  ODARS  Phase  IB  Test 
(Worstine,  1995,  p.  107). 

Review  of  Government-Owned  Survey  Administration  Software 


Because  of  unique  research  requirements,  the  government  has  developed  a  series  of 
software  systems  which  successively  approach  the  requirements  for  a  deployable  survey  admin 
capability. 

Project  LAMP'S  DRIVER  Program  (1981)  -  The  first  such  system  was  the  "DRIVER" 
system  for  Project  LAMP  (Learning  Abilities  Measurement  Program).  This  system  was 
designed  to  have  "drop-in"  task  lists  and  used  a  high-level  authoring  language.  One  could  easily 
specify  variations  from  screen  colors  to  total  session  protocol.  As  the  system  was  used  to 
measure  the  learning  ability  of  incoming  Air  Force  recruits,  the  "users"  were  assumed  to  span 
from  total  computer  illiterate  to  hacker. 

Experimental  Scale's  Computer-Administered  Surveys  (CAS,  1991)  -  For  the  purposes  of 
the  present  study,  such  a  government-created  software  system  was  already  under  development  in 
a  recent  AL/HRMJ  computerized  survey  software  and  scaling  project  (Albert,  et  al.,  1993; 

Phalen  &  Mitchell,  1993).  A  large-scale  data  collection  effort  was  completed  at  Lackland  AFB, 
TX.  These  data  have  been  analyzed  and  a  final  report  drafted  (Dittmar,  Hand,  Tucker  & 
Weissmuller,  1995).  The  software  system  used  in  this  project  required  a  hard  disk  in  order  to 
hold  three  floppy  disks  of  programs  for  each  individual. 

Computer- Administered  Occupational  Surveys  (CAOS,  1994)  Software  -  A  modified 
version  of  CAS  was  developed  specifically  for  the  present  study.  The  CAS  software  was 
modified  to  use  only  the  traditional  time-spent  scale  and  could  be  executed  from  a  single  double 
density  disk.  The  USAF  Job  Inventories  for  two  AFSs  for  this  study  were  programmed  into  the 
modified  software. 
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Functional  Relationships's  Occupational  Analysis  Surveys  (OASurv,  1995)  -  A  second 
Air  Force-owned  survey  system  was  deployed  in  November  1 995  for  data  collection  under 
another  task.  This  system,  the  Occupational  Analysis  Survey  System  (OASurv),  is  drawn  from 
the  same  source  code  library,  Computer-administered  Surveys  (CAS),  that  CAOS  used  as  its 
starting  point.  OASurv  complies  with  the  original  design  principles  which  targeted  potentially 
novice  users,  while  CAOS  moved  in  the  direction  of  advanced  features  for  more  computer- 
literate  users.  The  OASurv  model  is  considered  more  appropriate  for  operational  AF 
occupational  analysis. 


The  Present  Feasibility  Study 

A  key  issue  in  the  feasibility  study  was  to  determine  the  extent  to  which  operational  units 
within  the  Air  Force  could  support  computer-based  data  collection.  That  is,  how  available  are 
personal  computers  (PCs)  in  the  operational  Air  Force  to  support  administration  of  the  surveys. 
Further,  are  there  sufficient  computers  which  would  be  available  to  support  computer-based 
occupational  surveys.  A  second  issue  was  the  extent  to  which  currently  available  commercial 
computer-based  survey  software  packages  could  be  used  for  the  survey. 

Determine  Current  Field  Hardware  and  Software  Capabilities 

Clearly,  if  PCs  were  not  available  at  most  Air  Force  locations  or  units,  the  widespread 
conduct  of  computer-based  occupational  surveyings  would  be  severely  limited.  In  addition,  if 
computers  were  only  available  at  limited  sites,  then  Air  Force- wide  surveying  may  not  be 
practical  today.  Moreover,  if  military  or  civilian  software  was  already  available  which  would 
serve  as  an  operational  system,  then  no  adaptation  or  modification  would  be  needed  to 
operationally  implement  computer-based  surveys.  If  civilian  software  for  such  a  use  already 
existed,  examination  of  the  costs  and  time  involved  could  serve  as  a  baseline  against  which  any 
new  system  or  software  could  be  assessed. 

Assess  Computer  Availability  (Hardware)  -  A  survey  was  developed  to  determine 
existing  resources  available  to  support  computer-based  occupational  surveying.  The  results  from 
this  survey  were  analyzed  in  terms  of  the  overall  potential  for  conducting  computer-based 
occupational  surveys. 

A  pilot  test  was  also  conducted  at  a  few  AF  bases  within  or  near  San  Antonio,  Texas  prior 
to  mailing  diskettes  world-wide.  Personnel  from  AL,  AFOMS,  and  IJOA  visited  these  bases  to 
observe  the  administration  of  the  surveys.  Based  on  the  results  of  the  pilot  test  details  (see 
Section  4.0),  several  changes  were  made  to  improve  usability  of  survey  diskettes  and  several 
procedures  were  modified  prior  to  the  main  feasibility  study. 

Availability  of  Commercial  Computer-Based  Job  Survey  -  A  thorough  review  of  the 
literature  was  completed  to  identify  civilian  job  analysis  systems  which  use  PCs  for  survey 
administration.  A  preliminary  review  of  the  area  indicated  that  computers  are  often  used  for 
analysis  of  data,  typically  scanned  from  optical  or  mark-sense  score  sheets,  or  for  interfacing 
with  a  cumulative  data  base  or  set  of  norms,  as  with  the  PAQ/PMPQ  Enter-Act  system  (PAQ 
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Services,  1990),  which  permits  generation  of  norm-based  individual  and  group  analysis  reports. 
Other  job  analysis  systems,  such  as  the  Fleishman  Job  Analysis  System  (F-JAS),  use  scanable 
score  sheets  and  suggest  that  computer  administration  will  be  developed  sometime  in  the  future 
(Fleishman  &  Reilly,  1992).  Other  software,  specifically  designed  for  computer  surveying  had  to 
be  examined  in  more  detail. 


Government  Ownership  of  Software 

It  was  preferable  to  use  government-developed  or  -owned  software  for  this  study,  as 
noted  earlier.  Issues  of  site  licensing,  limitations  of  existing  software,  and  proprietary  rights,  in 
the  short  time  frame  planned  for  the  study,  made  it  impossible  to  properly  test  other  available 
commercial  software.  In  addition,  only  with  Air  Force-owned  or  -developed  software  could  an 
operational  program  be  implemented  that  would  be  flexible  enough  to  incorporate  future 
software  enhancements,  as  they  became  available  from  government  research  and  development 
programs  or  from  other  sources. 

Candidate  Career  Fields  for  Study 

A  number  of  career  fields  were  considered  for  study.  Criteria  for  consideration  were  (a) 
recent  USAF  Job  Inventories  completed,  (b)  Job  Interview  being  currently  administered  in  the 
field,  (c)  significant  populations  at  sites  in  the  CONUS,  USAFE,  and  PACAF,  and  (d)  sufficient 
population  sizes  for  study.  In  one  or  both  AFSs,  it  was  possible  that  some  individuals  might  take 
both  forms  and,  if  so,  these  would  be  compared  as  Time  1  -  Time  2  samples  for  reliability 
(stability)  assessment.  However,  one  or  both  AFSs  had  to  be  of  sufficient  size  so  that  a  new 
random  sample  could  be  generated  to  survey  about  800  -  1 ,000  cases. 

The  two  career  fields  chosen  for  the  study  were:  1)  AFS  3A0X1  (formerly  702X0), 
Information  Management,  and  2)  AFS  2A6X4  (formerly  454X3),  Aircraft  Fuel  Systems.  Based 
on  findings  from  the  survey  of  computer  availability,  it  was  expected  that  PCs  would  be  readily 
available  to  most  individuals  in  the  Information  Management  area,  since  their  work  normally 
involves  computers,  but  not  as  accessible  for  the  Fuel  AFS,  who  work  predominantly  in  a 
flightline  environment.  In  addition,  because  both  AFSs  were  spread  widely  across  the  Air  Force, 
some  generalizability  of  results  was  expected. 

Review  and  Debug  Computer-Based  Job  Inventories 

Experimental  survey  software  was  adapted  for  the  study.  Electronic  versions  of  the  Job 
Inventories  used  in  the  paper-and-pencil  field  administration  were  provided  by  the  AFOMS.  The 
Task  Inventory  Analysis  Requirements  Application  (TIARA)  system  files  for  both  AFSs 
included  background  sections  as  well  as  the  actual  task  lists.  The  task  lists  were  directly  uploaded 
and  edited,  but  background  questions  had  to  be  hand  entered  and  formatted. 

In  addition,  Task  Difficulty  (TD)  and  Training  Emphasis  (TE)  ratings  were  collected  via 
computer.  Separate  TE  and  TD  samples  were  also  generated,  with  minimum  overlap  with  the 
traditional  paper-and-pencil  administrations. 
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Selection  of  Random  Samples  of  AFS  Personnel 

Random  samples  of  personnel  to  receive  the  computer-based  method  were  selected.  This 
was  done  through  merging  and  matching  sample  files  from  the  paper-and-pencil  distribution  of 
each  survey  with  the  latest  Uniform  Airman  Record  (UAR)  files  to  develop  comparable  samples 
for  computer-based  survey  administration.  Where  possible,  individuals  receiving  the  paper-and- 
pencil  versions  were  purposefully  omitted  from  the  computer-based  survey;  however,  in  the 
smaller  AFS  (Fuels)  some  overlap  was  necessary. 

Once  samples  were  selected,  the  AFOMS  mailed  electronic  versions  of  the  surveys  and  a 
packet  detailing  procedures  for  completing  the  surveys.  An  E-mail  message  was  sent  from 
AFMPC  to  SCMs  involved  in  the  test,  requesting  their  participation  and  explaining  the  nature  of 
this  feasibility  study.  This  kept  the  objectives  of  the  test  open  and  above  board,  and  helped 
SCMs  observe  and  report  logistical  problems/issues  which  occurred  during  this  special  field 
administration. 


Uploading  Data  and  Analyze  Results 

Upload  Data  -  Upon  receipt  of  completed  disks  from  the  AFOMS,  I  JO  A  personnel 
compiled  the  data  into  separate  AFS  files  and  uploaded  the  files  to  the  AFOMS  computer.  Some 
new  software  was  developed  to  accumulate  individual  case  data  from  single  disks  into  a  common 
data  file  for  each  AFS,  and  a  special  program  reformatted  the  data  into  ASCII  CODAP  standards. 

Collect  Data  on  Logistical  Issues/Problems  -  In  addition  to  having  SCMs  administer  the 
computer-based  surveys,  they  were  also  asked  to  identify  difficulties  or  problems  they 
encountered  (which  differ  from  their  normal  survey  administration  activities),  how  they  thought 
participants  accepted  this  different  type  of  data  collection,  and  any  recommendations  they  might 
have  concerning  the  potential  operational  implementation  of  a  computer-based  survey  system. 
Some  of  these  questions  paralleled  items  asked  in  the  follow-up  survey  included  on  all  survey 
disks,  but  aimed  to  acquire  the  SCM  perspective  on  the  system,  particularly  through  open-ended 
comment  and  suggestion  items. 

These  data,  as  well  as  feedback  from  AFOMS  personnel,  served  as  a  basis  for  identifying 
critical  issues  for  operational  implementation  of  a  computer-based  survey  system.  Time 
estimates,  mailing  costs,  and  other  relevant  information  were  developed  from  AFOMS  estimates, 
to  serve  as  a  baseline  for  comparing  the  relative  cost  effectiveness  of  such  a  new  system. 

Analysis  of  Data  Equivalence  -  A  key  issue  in  this  feasibility  study  was  whether  or  not 
computer-based  survey  data  collection  was  functionally  equivalent  (not  necessarily  identical)  to 
data  collected  via  traditional  paper-and-pencil  methods.  One  criteria  was  the  distribution  of 
cases  across  job  types/clusters  in  a  joint  Comprehensive  Occupational  Data  Analysis  Programs 
(CODAP)  diagram.  If  cases  are  randomly  generated  and  data  collection  equivalent,  then 
computer-based  survey  cases  should  be  distributed  in  all  jobs,  and  should  not  cluster  just  with 
themselves.  That  is,  the  presence  of  a  job  type/cluster  totally  composed  of  computer-based 
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survey  respondents  would  suggest  that  data  collected  via  this  method  would  not  be  functionally 
equivalent  to  paper-and-pencil  collected  data.  Conversely,  relative  random  distribution  of 
computer-based  cases  across  the  job  types/clusters  of  a  joint  diagram  would  seem  to  substantiate 
that  the  data  are  probably  functionally  equivalent. 

In  addition,  analysis  of  follow-up  survey  questions  asked  of  computer-based  survey 
respondents  was  analyzed  to  determine  respondents  attitude  towards  the  computer-based  survey 
process.  Responses  of  members  of  the  two  AFSs  involved  were  compared,  to  determine  if  there 
were  AFS-specific  differences  in  how  personnel  reacted  to  computer-based  surveying.  One 
would  expect  that  Information  Management  personnel  would  have  less  difficulty  with,  and  more 
acceptance  of,  computer-based  survey  procedures,  since  they  use  computers  in  their  day-to-day 
work.  Aircraft  Fuel  Systems  personnel,  on  the  other  hand,  might  have  more  difficulty  with  or  be 
less  comfortable  with  computer-based  surveying  due  to  their  presumed  lack  of  ready  access  to 
PCs. 


Document  Feasibility  Study  and  Make  Recommendations 

The  study  plan  also  required  that  activities  conducted  under  this  feasibility  study  would 
be  documented,  and  that,  on  the  basis  of  the  results  analyzed,  the  study  team  would  develop 
recommendations  for  future  computer-based  occupational  surveys  in  the  Air  Force  (and  by 
extension,  the  other  military  services).  The  documentation  would  be  in  the  form  of  a  final 
technical  report  of  the  project. 

Logistical  issues  must  be  highlighted,  particularly  the  relative  costs  of  paper-and-pencil 
surveys  versus  computer-based  data  collection.  While  not  totally  comparable,  a  cost-per-copy 
would  be  calculated  as  a  prime  comparison  figure. 

Recommendations  were  also  to  be  made  with  regard  to  how  computer-based  surveys 
might  best  be  generated,  and  to  the  operating  systems  (“engines”)  which  might  be  most  efficient. 

Pilot  Test 

Prior  to  final  AF  distribution,  a  small  pilot  test  of  the  computer-based  occupational 
survey  system  was  conducted.  The  pilot  test  was  conducted  to  preclude  any  major  difficulties 
with  the  subsequent  full-scale  field  administration. 
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Initial  Pilot  Test  Results 


The  pilot  test  was  conducted  at  four  Air  Force  Bases  (AFBs)  in  Texas:  Brooks  AFB, 

Kelly  AFB,  and  Randolph  AFB  in  San  Antonio,  and  Laughlin  AFB  in  Del  Rio.  Eighteen 
personnel  performing  duty  in  Air  Force  Specialties  (AFSs)  2A6X4,  Aircraft  Fuel  Systems  and 
3A0X1,  Information  Management  were  selected  to  participate  in  the  test.  These  personnel  were 
asked  to  complete  Job  Inventory  or  Task  Difficulty  surveys.  The  floppy  disks  containing  the 
survey  programs  were  hand-carried  to  Survey  Control  Monitors  (SCMs)  at  Brooks,  Kelly,  and 
Randolph  AFBs  and  mailed  to  the  SCM  at  Laughlin  AFB.  SCMs  then  distributed  the  disks  to 
unit  training  managers  who  distributed  the  disks  to  survey  participants.  In  most  instances,  the 
floppy  disks  were  returned  to  the  SCMs  and  picked  up  by  a  member  of  the  IJOA.  A  total  of  1 7 
surveys  were  completed  and  returned;  one  individual  at  Kelly  AFB  had  just  retired  and  was  not 
available  to  participate.  This  represents  a  94%  return  rate,  which  is  as  good  or  better  than  normal 
AFOMS  sampling  for  most  specialties. 

Seven  verbal  recommendations  were  made  by  SCMs  and  survey  participants.  The  SCM 
at  Randolph  AFB  recommended  color  coding  be  used  to  distinguish  between  the  various  types  of 
surveys.  One  survey  participant  at  Randolph  AFB  made  two  recommendations:  1)  enable 
comments  to  be  made  about  tasks  when  completing  Task  Difficulty  surveys,  and  2)  add 
questions  designed  to  solicit  opinions  about  the  CAOS  program  at  the  end  of  all  surveys.  At  the 
time  of  the  pilot  test,  comments  about  tasks  and  the  CAOS  program  could  only  be  made  in  the 
job  inventory  surveys.  The  SCM  at  Brooks  AFB  recommended  the  surveys  be  made  available 
on  3.5"  floppy  disks,  since  the  majority  of  PCs  at  Brooks  have  3.5"  disk  drives  and  these  disks 
are  not  as  delicate  as  the  5.25"  disks.  Another  survey  participant  at  Randolph  AFB  suggested 
incorporating  the  capability  to  use  a  mouse  during  survey  administration.  Yet  another  survey 
participant  at  Randolph  AFB  recommended  policies  be  changed  to  require  survey  administration 
in  a  controlled  environment.  This  participant  completed  the  survey  in  his  duty  section  and  was 
constantly  interrupted  by  phone  calls  and  members  walking  into  his  section  seeking  assistance. 

Written  comments  received  from  personnel  completing  surveys  are  shown  below.  A 
feature  to  display  names  and  telephone  numbers  of  survey  participants  has  been  incorporated. 
This  information  was  not  previously  available  to  occupational  analysts. 
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Example  Comments 


Comments  Received  from  Participants  in  AFSC  2A6X4 
A433  is  J.  SMITH 

A433  1-  1 :  14  DEPOT  LEVEL  AIRCRAFT  FUEL  SYSTEMS  JOURNEYMAN 
A433  1-  1 :  45  THERE  ARE  NO  INSPECTIONS  TO  BE  PERFORMED  ON  AIRCRAFT  AT 
AFB.  THE  ONLY  INSPECTIONS  DONE  ARE  FOR  DAILY  USE  ON  POWERED 
AND  NON-POWERED  AGE  EQUIPMENT. 


Comments  Received  from  Participants  in  AFSC  3A0X1 
A078  is  K. ANONYMOUS 
A078  1-1:14  OPERATIONS 

A078  1-1:28  SUPERINTENDENT,  PROGRAMS  &  OPERATIONS 
A810  is  G.HERMES 

A810  1-1:22  MY  3A071  SKILLS  ARE  USED  SPARINGLY,  MOST  OF  MY  JOB  IS 
HANDLING  ANY  COMPUTER  PROBLEMS  THAT  OCCUR. 

A810  1-1:24  THE  JOB  I  DO  IS  VERY  IMPORTANT  TO  MY  ORGANIZATION,  IT  CUTS 
DOWN  TIME  OF  COMPUTERS  BY  90%.  MY  JOB  IS  VERY  SATISFYING 
A810  1-1:  33  NEED  TRAINING  ON  COMPUTER  TROUBLESHOOTING,  AND 
HARDWAREINSTALLATION 
A813  is  U.SAM 

A8 1 3  1-1:21  INFORMATION  MANAGEMENT  HAS  BEEN  GOOD  TO  ME  BUT  I’M 
READY  FOR  CHANGES.  I  DON'T  WANT  TO  SOUND  LIKE  INFO  MGT  IS  A  BAD 
CAREER  FIELD  BUT  I'M  READY  FOR  CHANGE— I'M  PRESENTLY  SEPARATING  AND 
TRYING  FOR  A  MEDICAL  SLOT  IN  RESERVE 

A813  1-1  :  32  THERE  IS  AN  ERROR  IN  THE  WORDING  OF  MY  QUESTION:  I  DO  HAVE 
NOT . 

A813  1-1 :3 3  I  AM  VERY  KNOWLEDGEABLE  IN  MOST  OF  THESE  COMPUTER  AREAS 
FROM  SELF-TEACHING  AND  OTHER  PEOPLE  WHO  HAVE  THE  EXPERIENCE. 

A81 3  2-1 :4  I  AM  ALL  FOR  THE  COMPUTER- ADMINISTERED  SURVEY  BUT  MAYBE  I 
SHOULD  HAVE  TAKEN  THE  SURVEY  IN  A  PLACE  OTHER  THAN  MY  OWN  OFFICE  - 
TOO  MANY  INTERRUPTIONS 

A8 1 3  2-1:9  THERE  WAS  NO  WAITING  BECAUSE  I  USED  THE  COMPUTER  ON  MY 
DESK  BUT  AS  STATED  EARLIER- WHEN  MY  CO-WORKERS  LEAVE  THE  OFFICE  I 
HAVE  NO  CHOICE  BUT  TO  ANSWER  THE  PHONES  AND  HELP  CUSTOMERS-IT  TOOK 
ME  ABOUT  1  HR  TO  DO  A  30  MIN  SURVEY 

Color  coding,  as  a  way  of  distinguishing  between  the  various  types  of  surveys,  was 
implemented.  In  addition,  questions  to  solicit  opinions  about  the  CAOS  were  also  inserted  at  the 
end  of  all  surveys.  The  use  of  3.5"  floppy  diskettes  will  be  considered  for  possible  future  surveys 
of  additional  AFSs.  One  criterion  for  the  pilot  test  was  to  ensure  surveys  could  be  administered 
on  all  existing  Air  Force  personal  computers  (PCs).  Therefore,  5.25"  low  density  diskettes  were 
used.  Since  many  Air  Force  PCs  do  not  currently  support  installation  and  use  of  a  mouse,  this 
capability  was  not  incorporated. 
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Pilot  Test  Findings 


Feedback  received  from  participants  and  observers  was  both  favorable  and  constructive. 
Participants  indicated  the  survey  programs  were  easily  executed  and  the  instructions  and  formats 
made  completion  of  the  surveys  quite  simple.  Participants  also  indicated  that  surveys 
administered  via  the  CAOS  software  required  significantly  less  time  than  those  administered 
manually.  Participants  reported  two  minor  errors  in  the  surveys  which  were  corrected 
immediately  following  identification. 

One  difficulty  was  encountered  that  invites  Air  Force  scrutiny  of  existing  data  control 
policies  and/or  procedures.  Current  policies  permitted  the  disks  distributed  to  participants 
assigned  to  the  Air  Intelligence  Agency  (AIA)  to  be  hand-carried  into  AIA  facilities  but  not  out 
of  the  facilities.  Participants  were  instructed  to  mail  the  disks.  The  costs  for  mailing  the  disks 
within  the  city  of  San  Antonio  exceeded  $3.00  per  disk.  This  situation  greatly  reduces  the  cost 
savings  to  be  realized  from  the  use  of  floppy  disks.  If  this  policy  stands,  recommend  participants 
be  scheduled  for  survey  administration  outside  of  the  AIA  facilities,  such  as  at  the  office  of  the 
SCM  or  the  base  Military  Personnel  Flight  (MPF)  testing  facilities. 

Interestingly,  there  were  no  instances  where  survey  participants  reported  that  a  computer 
was  not  available  for  use  in  this  pilot  test.  This  and  the  other  findings  of  the  pilot  test  were 
sufficiently  positive  that  the  full  field  test  administration  could  be  initiated. 

Full-scale  Field  Administration 
Initial  Administration 

A  total  of  two  thousand  two  hundred  5.25",  double-density  disks  were  produced  (1000 
job  inventories  per  specialty  plus  50  task  difficulty  and  50  training  emphasis  questionnaires),  at  a 
cost  of  about  $.56  per  copy  (not  including  shipping  costs).  These  disks  were  mailed  to  Air  Force 
bases  and  remote  locations  worldwide  by  the  staff  of  the  AFOMS,  using  their  normal  procedures, 
with  a  letter  requesting  quick  administration. 

Some  problems  were  encountered  in  the  initial  field  administration.  Some  disks, 
particularly  task  factor  rating  disks,  tended  to  "freeze  up"  the  system.  Telephone  contact  with 
survey  control  monitors  at  several  bases  led  to  identification  of  two  problems;  one  had  to  do  with 
the  version  of  MS-DOS  available  on  the  equipment  —  the  software  used  some  MS-DOS  4.0 
commands  not  available  in  earlier  versions.  The  second  issue  was  one  of  internal  memory 
required;  machines  with  512K  or  less  were  not  capable  of  executing  the  survey  administration 
program. 
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Readministration 


The  Air  Force  operations  community,  through  survey  control  monitors,  was  notified  of  a 
recall  of  all  task  factor  disks.  At  the  same  time,  suggestions  were  received  from  survey  control 
monitors  on  possible  administrative  and  system  improvements,  such  as  using  different  colored 
disks  for  each  type  of  survey  (task  inventory,  training  emphasis,  task  difficulty)  and  each  AFS. 
Task  factor  programs  were  modified,  the  follow-on  questions  added,  the  revised  system 
reproduced  on  3.5"  multicolored  disks,  and  disks  were  remailed  to  the  field  approximately  30 
days  after  the  recall. 

Of  the  2,000  job  inventory  disks,  only  about  1,900  were  actually  mailed,  since  some 
organizational  addresses  proved  to  be  obsolete  (particularly  some  of  the  support  bases  in  Europe 
for  geographically  separated  units).  Also,  a  decision  was  made  to  exempt  Lackland  AFB  from 
participation  since  they  had  participated  extensively  in  the  earlier  CAS  study. 

Processing  and  Uploading 

Upon  return,  all  disks  were  virus  checked  before  any  attempt  was  made  to  upload  data 
files.  Of  the  1900  disks  mailed,  19  (or  1%)  were  found  to  be  contaminated  with  some  type  of 
virus.  All  but  one  were  deactivated  and  survey  data  files  recovered. 

Virus-checked  disks  were  then  processed  to  upload  data  files  to  a  computer.  The  upload 
utility  program  handled  the  multiple  data  files  on  each  disk  sequentially;  this  software  was 
subsequently  modified  to  handle  all  in  an  integrated  process  which  included  entering  the  case 
control  number  onto  the  disk.  Disks  were  rejected  as  unusable  if  they  did  not  contain  key 
background  data  and  the  task  inventory  responses.  Of  the  1000  original  disks  per  AFS,  627  cases 
were  successfully  uploaded  for  the  Aircraft  Fuel  Systems  specialty  (62.7%  usable  return  rate)  and 
656  were  loaded  for  the  Information  Management  specialty  (65.6%  usable  return  rate). 

The  Task  Difficulty  and  Training  Emphasis  diskettes,  which  were  initially  withdrawn  and 
resent,  were  returned  at  a  substantially  lower  rate.  In  addition,  those  diskettes  which  were 
returned  were  incomplete  and  largely  unusable.  The  problem  with  most  diskettes  appeared  to  be 
the  “work  around”  used  to  overcome  limitations  of  the  survey  engine.  A  different  software 
engine  was  needed  for  task  factor  surveys  and,  in  fact,  had  recently  been  developed.  In  the 
future,  the  task  factor  capability  should  be  incorporated  into  standardized  occupational  survey 
software. 


Data  Analysis 

A  variety  of  analyses  were  performed  on  the  returned  data  or  restricted  subsets  of  the 

data. 


Time  1  -  Time  2  Reliability  -  For  the  Aircraft  Fuel  Systems  specialty,  there  were  not 
sufficient  potential  participants  to  avoid  duplication  between  the  recent  paper-and-pencil  survey 
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administration  and  the  present  computer-based  survey.  This  fortuitous  situation  created  a  very 
natural  opportunity  to  assess  a  repeated  measures  reliability,  at  least  for  those  cases  where  the 
individual  was  in  the  same  job  (i.e.,  had  not  transferred  to  another  unit  or  changed  job  title). 

The  assessment  of  whether  an  individual  was  still  in  the  same  job  was  a  subjective  one, 
based  on  available  background  data  from  the  two  survey  administrations.  Where  an  obvious 
change  had  taken  place  (major  reassignment  or  change  in  duty  title),  the  decision  was  reasonably 
clear  and  the  individual  was  excluded  from  the  reliability  assessment.  Where  no  major  change 
was  obvious,  the  individual  was  included  even  though  the  number  of  reported  tasks  performed 
might  have  changed  substantially. 

For  Information  Management,  a  totally-independent  random  sample  was  selected  since 
there  was  sufficient  population  available;  even  in  this  case,  however,  a  few  repeat  administrations 
were  found,  which  provided  at  least  a  small  sample  for  Time  1  -  Time  2  reliability  assessment. 
For  both  AFSs,  the  time  interval  was  12  to  18  months,  versus  the  usual  two  to  four  weeks.  The 
results  of  these  analyses  are  shown  in  Table  1  below. 

Table  1.  Time  1  -  Time  2  Reliability 


Aircraft  Fuels 
2a6x4 

Information  Management 
3a0xl 

Number  of  Repeated  Cases 

290 

31 

Number  in  Same  Job  Type 

124 

11 

After  12-18  Months 

Range  of  T1-T2  Correlations 

<N 

oo 

o 

.38  -  .75 

Average  Fisher  Z 

.67 

.70 

Average  Number  of  Tasks  At  Time  1 

175.50 

63.80 

Average  Number  of  Tasks  At  Time  2 

214.40 

92.10 

Difference 

38.90 

28.30 

Percent  increase 

22% 

44% 

Remarkably,  even  though  the  time  interval  between  administrations  is  quite  extended,  the  range 
of  individual  correlations  is  quite  comparable  with  a  series  of  Time  1  -  Time  2  reliability  studies 
accomplished  in  the  early  years  of  task-based  military  occupational  analysis  and  job  analysis 
studies  in  general  (Sellman,  1968,  pp.  244-245;  McCormick,  1979,  pp.  130-132). 

In  both  AFSs,  there  was  a  marked  increase  in  the  average  number  of  tasks  performed;  these 
increases  were  much  greater  than  anticipated.  While  some  increase  in  the  number  of  tasks 
performed  might  be  a  function  of  job  growth  over  time,  we  cannot  rule  out  the  possibility  of 
some  method  effect;  that  is,  that  this  increase  may  be  due,  in  part,  to  the  use  of  the  computer  as 
the  data  collection  method.  Earlier  studies  contrasting  computer  and  paper-and-pencil 
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occupational  surveys  revealed  a  similar  trend  with  computer-based  administration  (see  Hudspeth, 
Fayfich,  &  Price,  1990),  but  the  effect  (assessed  with  a  counterbalanced  experimental  design) 
was  on  the  order  of  six  to  nine  percent  difference.  The  differences  in  both  career  fields  used  in 
this  study  greatly  exceed  what  would  be  expected  from  method  effect  alone. 

One  possible  explanation  for  such  differences  may  be  the  result  of  having  the  computer  enforce 
the  two-stage  rating  process,  where  the  individual  first  checks  all  the  tasks  in  the  task  list  he  or 
she  performs,  and  then  do  a  second  pass  through  the  task  list  providing  a  time  spent  rating  for 
only  the  tasks  reported  as  performed  in  the  initial  pass.  With  the  computer-based  survey,  this 
procedure  is  rigidly  enforced;  only  those  tasks  checked  on  the  first  pass  are  displayed  for  the 
time-rating  pass.  In  paper-and-pencil  administration,  it  is  possible  for  a  respondent  to  check  and 
rate  in  one  pass  even  though  this  is  contrary  to  instructions. 

This  potential  explanation  deserves  further  study,  preferably  with  a  large  sample,  well-controlled, 
counterbalanced  experimental  design.  It  may  be  possible  with  such  an  experimental  study  to 
demonstrate  that  computer-based  survey  administration  yields  improved  occupational  data 
through  the  structured  enforcement  of  the  two-pass,  check  and  rate,  system. 

In  addition,  note  the  difference  in  the  amount  of  increase  between  the  two  specialties  involved  in 
this  feasibility  study.  It  would  be  worthwhile  to  expand  and  extend  the  results  of  this  feasibility 
study  by  studying  more  specialties  and  obtaining  enough  data  to  begin  to  understand  whether 
these  are  systematic  or  coincidental  findings. 

Comparability  Analysis  -  A  primary  question  to  be  answered  in  this  study  is  whether  computer- 
based  survey  administration  yields  the  same  or  a  different  picture  of  a  specialty.  One  way  to 
address  this  question  is  to  compute  the  correlation  between  the  overall  job  description  of  paper- 
and-pencil  cases  for  an  AFS  with  the  same  job  description  from  the  computer-based  survey. 

Such  correlations  could  be  calculated  on  the  percent  time  spent  (PTS)  vector,  the  percent 
members  performing  (PMP)  data,  or  the  rank  ordering  of  tasks  on  the  PTS  or  PMP  information 
(see  Table  2). 
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Table  2.  Correlations  of  Original  and  CAOS  Cases  (Facmat) 


Aircraft  Fuels 
2a6x4 

Information  Management 
3a0xl 

Percent  Time  Spent  (PTS) 

.97 

.98 

Percent  Members  Performing  (PMP) 

.98 

.98 

Rank  Order  PTS 

.98 

.96 

Rank  Order  PMP 

.98 

.97 

Original  N 

1145 

2442 

CAOS  N 

627 

656 

The  results  of  these  analyses  demonstrate  that  the  overall  picture  of  an  occupation  is  the 
same  for  computer-based  occupational  surveys  as  it  is  for  paper-and-pencil  survey 
administrations,  whether  assessed  using  PTS,  or  rank  order  statistics.  Yet  the  earlier  Time  1  - 
Time  2  analysis  indicated  that  the  computer-based  surveys  result  in  more  detailed,  perhaps  more 
complete  job  descriptions.  Clearly  there  are  more  analyses  which  can  and  should  be  done  with 
these  data,  and  more  experimental  studies  are  needed  to  clarify  exactly  what  is  happening  in  the 
computer-based  environment.  Additional  analyses  have  been  completed  comparing  job  and  task 
clustering  results  with  the  different  data  sets,  as  well  as  comparing  job-type  descriptions  for  the 
major  job  variations.  There  are  no  compelling  data  to  suggest  that  computer-based  job  data  are 
in  any  way  inferior  to  paper-and-pencil  data.  Results  indicate  that  the  computer-based  survey 
process  is  more  systematic  and  consistent. 

A  second  analysis  was  attempted  by  comparing  the  job  types  analyzed  in  the  initial 
occupational  survey  report  (OSR)  based  on  the  paper-and-pencil  survey  results  with  job  types 
found  in  the  subsequent  computer-based  survey.  A  comparative  analysis  of  the  two  surveys 
separately  indicated  that  all  major  job  types  were  found  in  both  samples,  except  for  one  small  job 
type  found  only  in  the  paper-and-pencil  study  (n  =  6  cases).  Again,  it  is  not  obvious  whether  this 
change  is  a  result  of  different  sampling  (size  of  samples  at  the  two  time  periods),  the  survey 
method  (paper  vs.  computers),  or  a  real  change  in  the  work  world  being  accomplished. 

Conclusions  -  Overall,  the  results  of  these  analyses  are  positive  and  suggest  that 
computer-based  surveys  are  as  good  as,  and  possibly  better  than,  traditional  paper-and-pencil 
survey  instruments.  Job  types  analyzed  will  be  generally  comparable,  with  the  possible 
exception  of  small  isolated  groups  who  represent  only  a  small  fraction  of  the  total  population. 

The  more  intriguing  possibility  is  that  computer-based  surveys  may,  in  fact,  result  in  better  and 
more  complete  results  through  systematic  enforcement  of  desired  survey  administration 
procedures.  With  no  external,  accepted  criterion  against  which  to  assess  the  two  methods,  there  is 
no  absolute  way  to  demonstrate  the  superiority  of  one  method  over  the  other. 
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Logistics  and  Cost  Analyses 


Among  the  major  objectives  of  the  study  were  analyses  of  the  availability  of  computer 
equipment  and  an  assessment  of  the  costs  and  logistical  problems  involved  with  computer-based 
survey  administration. 


Cost  Analyses 

To  address  the  cost  issues,  the  costs  of  the  current  survey  program  were  examined.  On  an 
average,  the  AFOMS  surveys  thirty-five  AFSs  per  year,  with  an  average  of  two  thousand 
incumbents  per  AFS.  The  manual  design,  development,  and  administration  of  survey 
instruments  have  proven  to  be  very  costly.  The  costs  for  AFOMS  to  print  a  single  survey 
instrument,  in  scanable  booklet  format,  ranges  from  $3.00  to  $10.00,  depending  on  the  format 
and  the  size  of  the  booklet.  For  example,  the  Fiscal  Year  1995  budget  for  printing  the 
occupational  survey  instruments  required  to  support  thirty-five  surveys  was  approximately 
$300,000.  This  budget  paralleled  the  actual  cost  during  Fiscal  Year  1994.  The  typical  cost  for 
mailing  a  single  average  survey  booklet  to  a  field  location  is  $520.  At  current  rates,  the 
approximate  cost  to  mail  booklets  for  the  35  surveys  conducted  annually  would  be  $18,200.  The 
total  estimated  costs  for  printing  and  mailing  survey  instruments  for  our  example  during  Fiscal 
Year  1995  is  $318,200  (excluding  the  personnel  man-hours  involved  and  overhead). 

Table  3.  Comparison  of  Annual  Survey  Costs  (based  on  35  surveys  per  year) 


Production 

Mailing 

Total 

Paper-and-Pencil  Surveys 

$300,000 

$18,200 

$318,200 

Computer-Based  Surveys 

$  39,200 

$18,200 

$  57.400 

Difference 

$260,800 

During  the  feasibility  study,  two  thousand  disks  were  produced  at  a  cost  of  $1,120,  or 
$0.56  each;  this  figure  includes  the  cost  of  labels  and  express  shipment  for  reproduction  of  the 
disks.  Costs  for  mailing  the  floppy  disks  were  virtually  the  same  as  for  mailing  booklets  (within 
$0.01).  The  production  and  mailing  costs  to  support  35  surveys  per  year  would  be  about 
$57,400.  This  represents  a  potential  annual  savings  of  $260,800  each  year  through  the  use  of 
computer-based  surveys  (see  Table  3). 

Additional  savings  may  be  feasible  if  disks  are  recycled  and  if  AFOMS  can  eventually 
procure  equipment  to  mass  produce  programmed  diskettes.  Such  savings  appear  to  be  highly 
significant  and  represent  a  significant  increase  in  the  economy  and  efficiency  of  the  program. 
Additional  savings  will  also  result  from  reduction  in  the  time  it  takes  incumbents  to  complete 
surveys  as  well  as  reduction  in  the  total  amount  of  time  required  to  conduct  an  occupational 
survey  project  (survey  development,  printing,  mailing,  data  scanning,  set  up,  analysis,  etc.),  it  is 
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difficult  to  estimate  the  latter  savings,  but  they  can  only  add  to  the  impressive  savings 
demonstrated  in  the  cost  analysis  provided. 

Respondent  Attitudes  and  Computer  Equipment  Availability 

Each  disk  administered  in  this  study  also  included  a  follow-up  survey  to  address  the 
availability  of  equipment  and  respondents'  attitudes  toward  computer-based  surveys. 

Attitudinal  Data  -  Survey  participants  indicated  they  much  preferred  the  computer-based 
surveys  to  the  “paper-and-pencil”  surveys  which  they  had  completed  previously  (see  Table  4). 
Participants  further  indicated  instructions  were  easy  to  follow  and  programs  were  easy  to  operate; 
few,  if  any,  problems  were  encountered  in  their  completing  the  job  inventories. 

Table  4.  Comparison  of  Respondent  Attitudes  (percent  who  slightly  agree,  agree,  and 
strongly  agree) 


Aircraft  Fuels 
2A6X4 

(N  =  Approx  649) 

Information 

Management 

3A0X1 

(N  =  Approx  697) 

Little  Difficulty  Locating  Computer? 

86% 

88% 

Computer  Functioned  Properly? 

87% 

92% 

Instructions  Easily  Understood?* 

77% 

82% 

Reasonable  Amount  of  Time  To  Complete?* 

41% 

57% 

Career  Field  Members  Can  Complete?* 

81% 

87% 

Prefer  Computer  Administration?* 

65% 

72% 

Computer  Survey  Was  An  Enjoyable  Experience? 

16% 

30% 

*Note:  in  A  Recent  U.S.  Army  ODARS  Study  (Worstine,  1995),  Members  of  Three  Military  Occupational 
Specialties  Were  Asked  Similar  Questions,  With  Comparable  Results,  Concerning  Completion  of  Computer-Based 
Training  Evaluation  and  Task  Performance  Surveys. 

Perhaps  the  most  significant  result  in  Table  4  is  the  comparability  of  responses  between 
the  two  specialties,  as  opposed  to  the  expected  lack  of  availability  of  equipment  or  computer 
capability  of  the  more  operational  AFS  (Fuels).  An  overwhelming  majority  of  job  incumbents  in 
both  specialties  had  no  difficulty  locating  a  computer,  operating  it  satisfactorily,  and  believing 
that  most  incumbents  in  their  field  could  do  likewise. 


The  major  differences  in  the  two  specialties  involve  their  perceptions  of  the  amount  of 
time  to  complete,  with  less  than  half  of  the  Fuels  persomiel  thinking  the  time  required  was 
reasonable  to  a  majority  of  the  Information  Management  personnel  who  felt  so.  The  difference 
here  may  be  a  function  of  the  differing  degrees  of  computer  familiarity  between  the  two 
specialties  since  Information  Management  personnel  routinely  work  with  computers  in  most 
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aspects  of  their  jobs.  Obviously,  the  more  flightline-oriented  Fuels  personnel  were  not  totally 
happy  with  the  amount  of  time  required,  even  though  their  responses  on  other  questions  were 
generally  positive. 

Only  a  small  number  of  individuals  answered  the  question  related  to  the  computer  survey 
being  enjoyable  positively.  Thus,  while  job  incumbents  are  able  to  find  equipment,  operate  it 
properly,  and  will  provided  requested  information,  we  cannot  expect  them  to  particularly  enjoy 
it.  They  do  it  because  it  is  necessary  and  important  to  the  Air  Force,  not  for  the  fun  of  it. 

Equipment  Availability  -  In  terms  of  the  equipment  available  to  job  incumbents  for 
completing  occupational  surveys,  the  data  collected  are  shown  in  Table  5. 

Table  5.  Comparison  of  Computer  Equipment  Availability  (percent  responding  yes) 


Aircraft  Fuels 

Information  Management 

2A6X4 

3A0X1 

(N  =  649) 

(N  =  697) 

Color  Monitor 

88% 

91% 

Mouse 

41% 

43% 

5.25"  Disk  Drive 

86% 

90% 

3.5"  Disk  Drive 

35% 

41% 

The  rather  surprising  findings  in  these  data  are  that  less  than  half  of  the  personnel  in  both 
specialties  reporting  having  a  mouse  available  on  their  computers,  which  infers  that  most  are 
operating  in  a  non- Windows  environment,  and  that  even  fewer  have  a  3.5 '  disk  drive  available. 
Data  from  the  Pilot  Test  had  suggested  that  3.5"  drives  were  widely  available  and  were  preferred; 
field  data  indicates  that  is  not  true  for  some  60  -  75  percent  of  the  Air  Force  population. 

These  findings  confirm  that  it  was  a  good  decision  to  use  the  lowest  common 
denominator  system  in  conducting  this  feasibility  study;  that  is,  5.25"  double-sided,  double¬ 
density  disks  as  opposed  to  the  more  advanced  high-density  drives.  By  using  the  less 
sophisticated  system,  more  individuals  and  units  were  able  to  participate  in  the  study. 

While  3.5"  drives,  mouses,  and  high  density  capabilities  are  becoming  much  more 
widespread  in  the  Air  Force,  as  are  Windows  applications,  there  is  no  universally  available 
hardware  system  (nor  any  standard  software,  even  though  some  organizations  have  mandated  the 
use  of  Microsoft  Word,  Excel,  and  PowerPoint).  There  is  typically  a  very  long  lead  time  before 
any  standard  systems  or  innovative  technologies  become  available  throughout  the  entire  Air 
Force,  simply  because  it  is  a  very  large,  geographically  and  organizationally  dispersed 
organization.  For  computer-based  surveying,  these  results  suggest  that  for  the  near  future,  at 
least,  the  use  of  low-end  systems  (5.25"  DS-DD  disks,  and  non- Windows  software)  should  be 
continued. 
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One  feasible  option  is  to  include  instructions  with  each  set  of  surveys  mailed  to  a  base  to 
permit  the  Survey  Control  Monitor,  or  other  knowledgeable  individual,  to  copy  the  survey 
software  locally  to  3.5"  disks,  if  that  will  facilitate  rapid  data  collection  (rather  than  having  job 
incumbents  have  to  hunt  for  a  computer  to  use).  For  most  AFOMS  purposes,  it  does  not  matter 
which  type  or  size  of  disk  is  used  to  transmit  data. 

Eventually,  this  issue  and  potential  problems  will  disappear,  as  computer  surveys  are 
transmitted  via  other  means  (milnet.  E-mail,  internet,  etc.),  as  has  been  suggested  by  the  U.S. 
Coast  Guard. 

Survey  Completion  Time  -  The  time  required  for  participants  to  complete  computer- 
based  surveys  was  approximately  one  half  of  that  typically  reported  by  participants  in  paper-and- 
pencil  surveys,  based  on  analysis  of  start-and-stop  times  recorded  on  a  selected  subset  of  disks 
(at  least  two  different  bases  for  each  AFS).  Participants  indicated  approximately  two  hours  had 
been  required  to  complete  paper-and-pencil  surveys.  Observations  by  persons  conducting  the 
pilot  test  and  by  Air  Force  personnel  during  the  administration  of  surveys  revealed  that,  on  an 
average,  approximately  one  hour  was  required  to  complete  a  computer-based  survey. 

Conclusions  and  Recommendations 

There  are  a  number  of  further  research  and  development  issues  which  need  follow-up 
action,  including  the  higher  average  number  of  tasks  performed  as  reported  by  the  computer- 
based  survey  respondents  and  what  that  implies  for  automated  job-  and  task-clustering.  There  is  a 
typical  problem  in  terms  of  which  result  is  an  acceptable  criterion;  are  we  just  attempting  to 
replicate  the  existing  data  analyses  or  do  the  computer-based  survey  results  yield  superior 
outcomes  (through  systematically  enforcing  the  desired  administration  procedures)?  Additional 
work  is  needed  to  clarify  these  and  other  findings  of  this  study. 

Clearly,  however,  it  has  been  demonstrated  that  deployment  logistics  pose  no  major 
barriers  to  operational  Air  Force  usage  of  computer-based  occupational  surveys,  and  that  major 
savings  are  possible,  particularly  through  avoidance  of  precision  printing  costs.  Minor  issues 
remain  in  aspects  of  electronic  deployment  and  data  reacquisition  (disk  mailing  vs.  E-mail,  etc.). 
This  means  that  attention  should  now  be  focused  on  developing  tools  to  facilitate  and  expedite 
automated  inventory  production.  Serious  evaluation  must  be  made  of  the  candidate  survey 
programs  (software  engine)  to  be  used  to  deliver  operational  Air  Force  Job  Inventories. 

Additional  support  software  is  needed  to  permit  integrated  virus  checking,  data 
conversion  (to  CODAP  standards),  and  file  consolidation.  With  new  tools  and  supporting 
methodologies,  Air  Force  occupational  analysis  appears  to  be  on  the  verge  of  a  quantum  leap  in 
reliability,  speed,  and  cost  effectiveness. 
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Appendix  A 


Study  2005  -  Aircraft  Fuels.  AFS2A6X4  (Formerly  454X3) 
Time  1  -  Time  2  Case  Correlations 


[Of  the  total  sample,  290  cases  matched  as  Time  1  -  Time  2  participants.  A  review  of  PRTVAR 
data  indicated  that  of  this  number,  only  124  were  in  the  same  job  type  after  the  14  -  18  month 
interval  between  the  two  survey  administrations.  Thus,  only  these  1 24  are  valid  Time  1  -  Time  2 
comparisons  in  terms  of  reliability  of  job  information.] 
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Mean 

SD 

Ntsk 

Correlation 
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.71 

1.61 

92 

1569 

1.14 

2.05 

136 

.67 

773 

1.06 

2.34 

108 

1204 

.24 

1.99 

205 

.57 

303 

3.33 

3.41 

273 

1614 

.67 

1.32 

145 

.50 

1068 

1.68 

3.01 

144 

1622 

1.82 

2.76 

187 

.60 

810 

1.91 

2.22 

387 

1730 

1.21 

1.78 

197 

.53 

738 

1.73 

2.70 

191 

1464 

1.77 

2.75 

211 

.66 

228 

.59 

1.33 

119 

1361 

.52 

1.22 

146 

.63 

1056 

2.36 

2.70 

293 

1519 

.94 

1.32 

323 

.48 

91 

.13 

.98 

11 

1321 

.28 

1.24 

37 

.54 

372 

1.34 

2.94 

108 

1329 

1.55 

2.86 

137 

.64 

1094 

.80 

1.63 

123 

1155 

.82 

1.57 

136 

.47 

254 

3.12 

2.91 

311 

1360 

3.94 

4.00 

298 

.59 

700 

1.50 

2.66 

167 

1665 

4.46 

3.55 

375 

.46 

718 

1.99 

2.51 

225 

1679 

2.93 

2.56 

318 

.48 

955 

2.08 

3.09 

192 

1219 

2.52 

3.16 

264 

.66 

883 

1.10 

2.05 

152 

1544 

1.63 

2.14 

250 

.57 

448 

.90 

1.82 

127 

29 
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.56 

1.16 

142 

.45 

560 

1.77 

2.65 

211 

1158 

1.78 

2.62 

239 

.63 

855 

.83 

1.67 
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1318 
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1.69 

132 

.59 
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75 
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2.13 

135 

.50 
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1.76 

2.07 

300 

1301 

2.25 

2.26 
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563 
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2.02 

113 
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1.13 

1.88 
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.66 

394 
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2.49 

211 

1257 

2.29 
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232 

.59 
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2.06 
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1.89 

205 

.49 

261 

2.14 

2.72 

227 

149 

13.38 

3.64 

288 

.69 

990 
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1407 
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2.67 
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3.08 
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1466 
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320 
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1.33 

2.18 
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3.35 

329 
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.49 

1.19 
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.43 

1.14 
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.66 
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.51 

1.72 

48 

1324 

1.76 

2.58 
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.41 

701 
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2.07 

112 

1248 

.74 

1.72 

96 

.51 
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1.44 

2.64 
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1390 

1.90 

2.91 
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.67 

721 

.63 

1.49 

138 

1676 

.78 

1.05 

317 
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741 

1.28 

2.47 
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1468 

1.26 

2.20 

137 

.51 

722 

2.49 

3.26 

227 

1477 

1.77 

2.39 

244 

.57 

30 


158 

1.09 

2.07 

135 

1353 

1.71 

2.38 

189 

.61 

68 

2.65 

2.58 

287 

1276 

1.84 

2.33 

222 

.45 

296 

.99 

1.88 

151 

1449 

1.38 

2.16 

192 

.59 

740 

1.59 

2.49 

188 

1673 

2.09 

2.46 

234 

.57 

41 

1.11 

2.06 

126 

1538 

2.27 

3.30 

191 

.64 

728 

1.44 

2.54 

160 

1674 

1.60 

2.38 

259 

.60 

569 

2.30 

3.28 

238 

1391 

2.29 

3.06 

241 

.72 

872 

1.72 

2.45 

224 

1541 
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1.35 
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.65 
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2.21 

2.57 
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1.84 

2.44 
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950 

1.60 

2.56 
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1217 

2.11 

2.45 

284 

.63 
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1.55 

2.78 
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1.82 

2.21 

236 
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2.03 

305 

.58 
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1.69 
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1552 
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1.37 
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.45 
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1.13 

1.80 
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1198 

2.54 

3.04 

318 

.48 

822 

2.58 

3.05 

287 

1306 

2.62 

2.81 

314 

.82 

570 

2.70 

3.12 
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1442 

1.90 

2.28 

244 

.77 

657 

1.48 

2.40 
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1151 

1.55 

2.22 

191 

.71 
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2.08 

2.95 
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1749 

1.27 

2.05 

193 

.62 

53 

1.45 

2.08 

201 

1654 

1.82 

2.38 

214 

.67 

173 

.95 

2.07 

113 

1597 

1.64 

2.36 

212 

.55 

1126 

2.15 

2.56 

244 

1511 

1.74 

1.81 

320 

.62 

237 

1.91 

3.00 

169 

1337 

1.18 

1.85 

183 

.74 

67 

1.12 

2.64 

88 

31 


1478 

1.12 

2.55 

102 

.66 

858 

1.07 

1.86 

166 

1315 

1.85 

2.12 

304 

.32 

152 

1.75 

2.65 

185 

1317 

2.23 

2.36 

295 

.57 

770 

1.32 

2.45 

147 

1734 

1.86 

2.15 

310 

.42 

272 

1.03 

2.07 

112 

1566 

1.58 

2.40 

174 

.53 

24 

1.73 

3.23 

134 

1695 

1.41 

2.31 

165 

.58 

153 

.88 

1.79 

175 

1352 

2.43 

2.56 

316 

.51 

504 

1.25 

2.02 

170 

1254 

1.38 

2.16 

190 

.41 

1022 

1.86 

2.59 

205 

1550 

2.17 

2.45 

271 

.65 

60 

2.27 

3.00 

231 

1458 

2.60 

2.84 

268 

.56 

1076 

.85 

1.77 

109 

1711 

1.41 

2.15 

176 

.64 

255 

5.35 

3.33 

428 

1362 

3.66 

2.48 

398 

.45 

457 

3.28 

3.93 

227 

1568 

1.81 

2.45 

197 

.55 

929 

1.77 

3.35 

137 

1175 

1.57 

2.52 

215 

.70 

739 

1.52 

2.49 

167 

1747 

2.13 

2.48 

236 

.59 

705 

1.57 

2.40 

171 

1669 

2.20 

3.28 

199 

.62 

64 

2.29 

2.46 

266 

1661 

2.48 

2.43 

306 

.59 

59 

3.08 

3.81 

232 

1643 

1.91 

2.42 

232 

.63 

842 

1.50 

2.27 

174 

1416 

1.60 

2.33 

196 

.50 

947 

1.17 

2.26 

144 

1179 

1.61 

2.73 

176 

.67 

177 

1.72 

3.25 

128 

1457 

2.44 

3.67 

190 

.51 

1090 

1.89 

3.15 

168 

1523 

3.34 

3.47 

307 

.60 

47 

1.28 

1.82 

218 

1662 

1.61 

2.32 

186 

.71 

32 


290 

2.43 

3.64 

174 

1645 

2.29 

3.18 

289 

.51 

949 

2.03 

3.27 

165 

1216 

1.46 

2.79 

133 

.70 

169 

3.14 

2.43 

520 

1455 

2.36 

2.59 

263 

.44 

826 

1.21 

2.38 

157 

1296 

1.67 

2.08 

280 

.54 

298 

.40 

1.30 

60 

1447 

1.79 

2.40 

198 

.30 

1  o  ^ 

1  J>J> 

1.81 

2.49 

228 

1471 

2.14 

2.62 

237' 

.53 

147 

1.78 

3.11 

154 

1340 

1.77 

2.93 

203 

.61 

77 

.68 

1.56 

112 

1556 

.59 

1.63 

121 

.71 

925 

2.01 

2.62 

236 

1180 

2.59 

2.92 

254 

.69 

140 

1.04 

1.92 

140 

1561 

1.64 

2.69 

184 

.61 

308 

1.74 

2.24 

234 

1703 

1.89 

2.60 

221 

.52 

1041 

.41 

1.44 

66 

1336 

.49 

1.61 

55 

.46 

1145 

1.13 

1.83 

173 

1192 

1.92 

2.30 

239 

.67 

752 

1.11 

2.05 

160 

1726 

1.75 

2.47 

203 

.34 

652 

.65 

1.46 

130 

1721 

.85 

1.64 

161 

.65 

287 

.11 

.72 

13 

1612 

.59 

1.76 

59 

.45 

30 

1.01 

2.15 

120 

1768 

1.55 

2.25 

197 

.59 

840 

1.65 

2.39 

217 

1759 

2.06 

2.79 

254 

.57 

5 

1.59 

2.29 

192 

1548 

1.81 

2.53 

230 

.58 

653 

2.30 

2.89 

255 

1156 

2.00 

2.29 

264 

.59 

757 

1.27 

2.24 

140 

1402 

.67 

1.90 

64 

.60 

717 

.60 

1.37 

134 

1465 

.59 

1.28 

142 

.57 

154 

1.58 

3.00 

140 

33 


1342 

2.03 

3.18 

188 

.63 

121 

1.08 

2.43 

120 

1598 

1.78 

2.51 

196 

.46 

540 

.58 

1.57 

69 

1147 

.95 

1.68 

140 

.44 

253 

.75 

1.96 

86 

1359 

1.08 

2.25 

130 

.47 

811 

2.00 

2.37 

256 

1717 

1.03 

1.52 

228 

.51 

75 

.25 

1.15 

27 

1234 

.27 

1.27 

25 

.73 

422 

1.25 

2.42 

147 

1166 

1.81 

2.52 

197 

.65 

397 

2.97 

2.65 

329 

1251 

2.10 

2.23 

291 

.67 

165 

1.01 

2.13 

123 

1641 

2.38 

2.67 

263 

.46 

279 

2.06 

2.47 

240 

1733 

2.50 

2.74 

292 

.52 

747 

1.41 

2.34 

179 

1558 

1.75 

2.32 

211 

.65 

792 

1.10 

2.18 

129 

1705 

1.81 

2.60 

216 

.64 

544 

.31 

1.35 

35 

1229 

.26 

1.35 

26 

.67 

748 

1.00 

1.97 

118 

1687 

1.19 

1.89 

177 

.68 

820 

.22 

.83 

62 

1366 

.34 

1.13 

68 

.53 

931 

1.74 

2.32 

209 

1183 

2.53 

3.20 

251 

.64 

252 

2.33 

2.47 

263 

1358 

2.17 

1.90 

335 

.58 

936 

1.29 

2.37 

155 

1173 

1.65 

2.35 

272 

.67 

Range: 

.30  -.82 

Average  Fisher  Z  for  124  correlations: 

.67 

Mean  number  of  tasks,  Time  1 : 

175.51 

Mean  number  of  tasks,  Time  2: 

214.40 

Difference 

38.89 

(=  22%  Increase  in  Average  Number  if  Tasks  Performed) 
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Appendix  B 


Study  2006  Information  Management,  AFS  3A0X1 
Time  1  -  Time  2  Case  Correlations 


[Of  the  total  sample,  only  31  cases  matched  from  Time  1  to  Time  2.  The  time  interval  was  from 
12  to  18  months  between  the  two  administrations.  Review  of  background  information,  such  as 
number  of  personnel  supervised,  job  title,  base  and  unit  of  assignment,  etc.,  revealed  that  only  1 1 
of  these  31  cases  remained  in  the  same  job  type  and  thus  were  a  suitable  sample  for  assessment 
of  the  stability  of  job  descriptions.] 


Case 

Mean 

S.D. 

Ntsk 

Correlation 

195 

.14 

.78 

30 

2697 

.06 

.56 

15 

.61 

2001 

.86 

2.35 

90 

2490 

1.81 

2.79 

224 

.57 

1008 

.91 

2.10 

117 

2717 

.90 

1.92 

129 

.55 

2403 

.51 

1.61 

67 

2532 

.59 

1.68 

80 

.43 

725 

.40 

1.46 

69 

3010 

.72 

2.00 

97 

.73 

1247 

.23 

1.21 

27 

2443 

.41 

1.62 

46 

.66 

1065 

.38 

1.51 

52 

2677 

.41 

1.42 

66 

.60 

1608 

.71 

1.92 

104 

2608 

1.01 

2.30 

137 

.75 

469 

.34 

1.30 

60 

2995 

.25 

1.02 

52 

.66 

929 

.30 

1.21 

55 

2556 

.95 

2.15 

131 

.38 

1729 

.23 

1.22 

31 

2939 

.19 

.97 

36 

.56 

Range:  38 -.75 

Average  Fisher  Z  for  124  correlations:  .70 

Mean  number  of  tasks,  Time  1 :  63.82 

Mean  number  of  tasks,  Time  2:  92.09 

Difference  28.27 


(—  44%  Increase  in  AverageNumber  of  Tasks  Performed) 
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